Hyperspherically regularized networks for self-supervision

Bootstrap Your Own Latent (BYOL) introduced an approach to self-supervised learning avoiding the contrastive paradigm and subsequently removing computational burden of negative sampling associated with such methods. However, we empirically find that image representations produced under BYOL's self-distillation are poorly distributed in representation space compared This work demonstrates feature diversity enforced by losses is beneficial uniformity when employed BYOL, as such, provides greater inter-class separability. Additionally, explore advocate use regularization methods, specifically layer-wise minimization hyperspherical energy (i.e. maximization entropy) network weights encourage uniformity. We show directly optimizing a measure alongside standard loss, or regularizing networks BYOL architecture minimize neurons can produce more uniformly therefore better performing for downstream tasks.